DOCUMENT RESUME 



ED 353 224 



SP 034 221 



AUTHOR Pellow, Randall A.; Kuhns, Polly 

TITLE Program Quality Assessment in Pennsylvania. 

PUB DATE Sep 92 

NOTE 57p. 

PUB TYPE Reports - Research/Technical (143) 

EDRS PRICE MF01/PC03 Plus Postage. 

DESCRIPTORS Academic Ability; Admission Criteria; ^^Educational 
Assessment ; ^'^Educational Qual i ty; Educat ion Maj ors ; 
Elementary Secondary Education; Evaluation Criteria; 
^Evaluation Methods; Higher Education; Institutional 
Evaluation; **Trogram Evaluation; Student 
Characteristics ; Summative Evaluation; ^Teacher 
Educat ion Programs 

IDENTIFIERS External Evaluation; ^Pennsylvania; ^Quality 
Indicators 



ABSTRACT 

Pennsylvania does not have a legislatively mandated 
program in place £or assessing teacher education programs in 
institutions of higher education. This study was conducted to examine 
how Pennsylvania's public and private universities and colleges 
assess the quality of their teacher education programs. Deans and/or 
chairs of teacher education programs from all higher education 
institutions in the state were contacted by the Pennsylvania Academy 
for the Profession of Teaching. Participants (23 institutions), 
selected at random, responded to a survey instrument designed to 
investigate assessment procedures used in determining quality of 
teacher education programs. Findings indicate: scholastic aptitude 
test scores and high school class standing are the two largest 
determinants for accepting students; internally derived factors are 
perceived to be more important than externally derived processes; and 
greater emphasis has been placed on more traditional^ quantitative 
measures than on qualitative indices. Tabular results, 
recommendations, and an extensive bibliography are included. 
Appendices consist of copies of the survey instrument , the letter 
introducing the project, responses to the survey instrument; and 
comments from respondents. (LL) 
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THE PROBLEM 



I nt r odu ct i o n 

During the last decade, there have been unprecedented 
efforts zo strengthen this nation's schools. Agitation 
throughout the educational arena for what is now popularly 
known as the school reform movement was initiated by the 
landmark publication h^]i^LLQ.n^hL~EX3.k (1983), and shaped 
more recently by reports from the Holmes Group (198 6), z'r.e 
Carnegie Foundation (1988), and the burgeoning literature. 

ScrTie critics in basic education argued for changes to 
occur within the schools (K-12) . Smaller class size and 
more instructional resources to enhance the quality of 
teaching, rather than expanding the teacher's initial 
preparation, were recommended by many. 

Seme reformers such as the Holmes Group pleaded for 
political/programmatic changes at institutions of higher 
education. They advocated post ba ccalaureate teacher 
preparation and the enhancement of the professional status 
of teachers. They discerned a "confederation as a more 
viable strategy for improving teacher education than single 
institutions moving independently of one another . . . 
(Howey and Zimpher, 1989, p. 3) and a hierarchial career 
ladder that would acknowledge different professional levels 
In addition , a manifestation of post baccalaureate training 
was the politically mandated induction or mentoring program 



Sorr.e reforiners advocated revisions and extensions to 
the undergraduate program focusing more upon curricula 
changes. These critics emphasized changing the nature of 
pedagogy and teaching effectiveness^ and redesigning general 
education as well as professional core courses. 

Other critics argued for a fifth year of training 
extending teacher training programs into graduate level 
work. The American Association of American Colleges for 
Teacher Education (AAACTE) identified seven alternative 
paradign^.s for extended teacher preparation programs (Howey 
and Zimpher, 1989, p. 4). 

Howey and Zimpher (1989) offered an overview of the 
current issues over teacher education. They succinctly 
amalgamated the dichotomy between basic and higher education 
by citing, "Recent concerns over our nation's schools and 
the quality of teaching within them have translated into 
questions of who prepares our teachers, in what manner, and 
how we 11?" (p . 1 ) . 

The public education system in the United States is a 
huge enterprise employing over two million teachers in 
eighty-three thousand schools (vTohnston and Others, 1989, 
p. 8). Whether one's concern is basic education (schools 
housing grades K-12), teachers, or teacher education 
programs in higher education, the permutations of variables 
to be explored makes for an exceedingly complex problem. 
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Indeed^ there is so much fervor and agitation in the 
literature over teaching and teacher education that terms 
such as "brouhaha"^ "firefight"^ and "shootout" were used to 
characterize the dissension within the reform movement. 

S t at ement o £ the P rob lem 

Pennsylvania does not have a legislatively mandated 
program assessment in place. Therefore^ the purpose of this 
study was to determine how Pennsylvania's institutions of 
higher education assess the quality of their teacher 
education programs.' More specifically, this study answered 
the following questions: 

1. What sources are used for students' admissions into 
universities and colleges in Pennsylvania? 

2. What processes are used within institutions of 
higher education to evaluate teacher education 
programs? 

3. What measures of students' abilities are 
used to evaluate quality in teacher education 
pr ograms? 

4. What student summative assessment data are used to 
evaluate teacher education programs 

5 . What factors are perce ived to contribute to the 
overall quality of teacher education programs? 

6. What external review agencies are used to evaluate 
teacher education programs? 



DESIGN OF THE STUDY 



Methodology 

Many states throughout the nation have mandated 
assessment programs for institutions of higher education. 
Pennsylvania is not one of the states where such a program 
has been legislatively determined (Cooper, letter Aug, 6, 
1991). Due to the number of institutions in Pennsylvania, 
the large number of Instructional I Certificates (issued 
yearly) and the litany of research, tne significance of 
assessment, as related to program quality of teacher 
education, has become paramount. 

Therefore, it was the intent of the researchers to 
examine assessment issues directly related to Pennsylvania 
institutions of higher education. Due to the multiple 
dimensions associated with assessment, it was determined by 
an Advisory Committee to the Academy that Phase I of this 
project would be to determine the ways colleges and 
universities in Pennsylvania assess the quality of teacher 
education programs. The additional phases needed to further 
investigate this issue have not been determined. Therefore, 
this section addresses the methodology utilized to carry out 
the investigation of Phase I. 



Sites and Subjects 

The state of Pennsylvania is the location of a myriad 
of institutions of higher education. Within its boundaries 
are 85 private institutions^ five state-related/aided 
institutions and 14 state institutions. During the 1290-51 
year^. these institutions granted 10^ 269 Instructional I 
certificates. In order to investigate the quality of these 
teacher education programs^ a sample of university types was 
chosen as the units of investigation. These institutions 
represent approximately 80% to 85% of teachers being 
certified on an annual basis. All 14 universities of the 
SSHE^ 5 of the s tat e- relat ed/ai ded institutions and 11 of 
the private institutions were targeted. Using che table of 
random numbers^ 11 private colleges or un ivers it i^"ns were 
randomly selected from a total population of 85. The deans 
and/or chairs of the teacher education programs from the 
selected universities or colleges were the respondents of 
the survey instrument . 

Instrumentation 

The survey research method was chosen as the means to 
investigate the assessment procedures used to determine the 
quality of teacher education programs in Pennsylvania. The 
survey instrument reflected the processes used by colleges 
and universities to assess quality from both internal and 
external sources. These sources were selected from 
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pertinent college catalogs and from the current literature 
on assessment procedures in higher education. 

In total, the survey consisted of twenty-one items and 
followed questionnaire and opinionnaire formats. The survey 
was subdivided into five distinct parts with each one 
containing multiple-itemed questions. Part I pertained to 
factors that related to teacher education program 
evaluation. The lesponses were categorized according to 
elementary and secondary programs. Part II addressed 
admissions criteria into colleges and universities and 
criteria for entrance into teacher education programs. Part 

III was concerned with student teaching prerequisites. Part 

IV dealt with institutional information. Part V referred to 
the perceptions of the respondents and their perceived 
opinions of their faculty on factors that contribute to the 
quality of teacher education programs. The survey also 
contained space for comments. 

A large portion of time regarding survey construction 
was spent formating the design. Applicable columns were 
computer drawn and labeled accordingly. This effort was for 
the purpose of visual clarity and organization. 

Validation of the survey instrument was s ought from 
experienced professionals. They were: Dr. Judy Brought 
Chair of the Educational Administration Department of a 
private liberal arts college; Dr. Audrey Sprenger^ former 
Director of an NCATE evaluation team; Dr. Robert Bartos^ 
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Dean of Education and Human Services at a SSHE institution/ 
Dr. Walt Konetschni, Chair /Prof essor of Sociology; Dr. Mary 
Cooper^ former Project Director of the Pennsylvania Academy; 
and an Advisory Committee. Revisions were made according to 
suggestions. Finally^ the instrument was titled "Program 
Quality Assessment Survey in Teacher Education in 
Pennsylvania." See Appendix A. 

Procedures 

Permis sion was granred from the Pennsylvania Academy 
for the Profession of Teaching on June 3, 1991 to conduct 
the study. Prior to approval, a proposal was submitted 
which detailed the goals of the project. The proposal 
delineated and calendarized the specifics of the 
investigation using the context^ input, process, and product 
(CIPP) evaluation model. The model specifically outlined 
the objectives and time frames necessary for the study's 
completion . 

From the onset, the researchers worked directly with 
the Executive Director and Project Director of the Academy, 
Dr . Susan Arisman and Dr. Mary Cooper, re spect ively , and 
members of the Pennsylvania Academy's Advisory Committee. 
Two meetings were held with this committee to discuss the 
intent of the research and to validate the assessment 
i nstrument . In add it ion , sever a 1 ot her meet in gs we re 



conducted with the Project Director during the construction 
of the instrument^ data collection, and data analysis. 

Following the construction, validation, and printing of 
the ins::rument, the surveys were sorted and preceded for 
mailing. The coding occurred for identification so that 
follow-up telephone interviews might be arranged, if 
necessary. A cover letter from the Academy was written and 
sent along with the survey to the identified institutions. 
(See Appendix B) . The coding and mailing was handled by zr.e 
Academy for efficiency, cost, and anonymity. The surveys 
were to be returned to the Academy by mid -Sept ember , 1991. 

Prior to analyses, the questions were categorized 
according to process and product aspects of teacher 
education programs. The categorization was done 
individually by the researchers and the Project Director, 
and then, ranked jointly in a following meeting. There was 
high agreement among the items. If calculated, the research 
team would have achieved high interrater reliability. 
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ANALYSIS OF DATA 

Method o£ Analysis 

In September 1991, The Program Quality Assessment 
Survey Instrument was sent to fourteen SSHE institutions, 
five state-relat ed/aided institut ions, and eleven randomly 
chosen private institutions. Survey responses were received 
from 23 of the 30 institutions that were sent surveys for a 
total return rate. Of the 23^ ten were from the State 

System cf Higher Education, three from the 

state-related/aided institutions, and 10 from the private 
institutions. There were 21 institutions with elementary 
teacher education programs and 23 institutions with 
secondary teacher education programs. Data were tabulated 
and analyzed collectively to preserve the anonymity of the 
respective institutions. 

Working under the direction of Dr. Tom Austin, 
Stat is tic j.an and Professor of Criminal Justice at 
Shippensburg University, a graduate assistant created a 
codebook for tallying the data on SPSS" Statistical Program. 
There was a total of 21 items plus a "Comments" section to 
be analyzed. Because of the dichotomy between elementary 
and se condary programs and the multi -responses for some 
itemS/ responses were coded and categorized ?.c cordi ngly . 
Thus, there was a total of 110 items that were analyzed. 
Data were tabulated and analyzed according to the number of 
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responses, the percentage of respondents^ and mean scores 
(where applicable). (See Appendix C). 

Results o£ Oata Analysis 

Many of the results are reported using tables. 
However, not all of the significant data were placed in 
tabular format. Thus^ the results reflect information 
extracted from the tables and i nf ormat ion expr essed 
separately in the narrative. In addition, such information 
is presented sequentially to address the questions upon 
which this study was based. 

Table One cites admission criteria reported by 
colleges and universities in Pennsylvania. Such findings 
reflect the dichotomous structure of the sample and 
therefore separately address elementary and secondary 
programs. Of the total number of institutions (n=^-23) that 
responded, 21 had teacher education programs in elementary 
education and all 23 had secondary programs. 

Both programs showed that the S£ili2l^stj.£_Aci:jLtiidfi_iI^^ 
and high school class standing were the two largest 
determinant s for accepting student s into Penns ylvan ia 
col leges and universities. Mo re specifically, 87% of the 
secondary teacher education programs cited these two 
criteria as a priority whereas in the elementary programs, 
86% and 81% respectively, were reported. Another admission 
criterion, high school grade point average, was used by 12 
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Table One 

Admissions Criteria Into Universities and Colleges 





Elementarv 


Second?/y 




(n=21) 


% 


(n=23) 


% 


Scholastic Aptitude Test 


1 8 


86% 


20 


87% 


High School Class Standing 


1 7 


81% 


20 


8 7% 


High School Grade Point Average 


1 2 


57% 


1 5 


65% 


High School Portfolios 


4 


19% 


4 


1 8% 


Admissions Interview 


8 


38% 


8 


35% 



Table Two 

Internal Processes Used to Evaluate Teacher Education Programs in PA 





Elementary 


Secondary 




(n=21) 


% 


(n= 


23) % 


Faculty Committee Evaluation 


1 9 


90% 


21 


91 % 


Evaluation Policy Guidelines 


1 5 


71% 


1 7 


74% 


Faculty Evaluations of Quality 


1 6 


76% 


1 8 


78% 


Student Evaluations are Used 


1 9 


90% 


21 


9 1 % 
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institutions (57%) with elementary programs and 15 
i TiStitutions (65%) with secondary programs. Eight 
institutions (35%) with secondary programs and 8 
institutions (38%) with elementary programs used admissions 
interview as an additional criterion for entrance, while 
only 18% (n = 4) and 19% (n = 4 ) of the institutions, 
respectively, utilized high school portfolios. 

Additional data regarding SAT scores were collected. 
Twelve institutions (52%) reported cutoff scores ranging 
between 300 and 1199. Nine institutions (39%) chose the 
response "not applicable" which possibly indicated the use 
of SATs but having no cutoff score. 

Regarding admissions into teacher education programs, 
most colleges and universities had separate entrance 
requirements. Only 22% of the secondary programs and 14% of 
the elementary programs used a 2.0 - 2.49 grade criterion, 
whereas 74% and 76% respectively, utilized a grade point 
average of 2.5 or higher. In addition, 14 elementary 
programs (67%) and 16 secondary programs (70%) required 
"approval by the Dean of Education." Disparity of results 
existed with an "interview committee;" only 5 schools with 
elementary programs (24%) used them whereas 16 schools with 
secondary programs (76%) found them to be necessary. 

Table Two demonstrates those assessment processes used 
within institutions to evaluate teacher education programs. 
"Faculty committees" and "student evaluations" were used in 
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Table Three 

Measures of Student Abilities Used 
in Program Evaluation 



Secondary 





(n=21) 


% 


(n=23) 


"•0 


Organize or formulate curriculum 


1 9 


90% 


2 1 


9 1 °'o 


Make subject matter comprehensible 


20 


95% 


21 


91% 


Adapt content and instruction to learners 


20 


95% 


22 


96% 


Assess learning 


20 


95% 


22 


96% 


Use assessment to organize instruction 


20 


95% 


22 


96% 


Work with students of diverse backgrounds 


1 9 


90% 


20 


87% 


Set goals for professional development 


1 4 


67% 


1 6 


70% 


Table Four 










Summative Assessment Data Used in Program Evaluation 








Elementary 
(n=21) % 


Secondary 
(n=23) % 


graduation ratios 


12 


57% 


1 5 


65% 


employment ratios 


1 7 


81% 


1 9 


83% 


student perfonnance on exit exams 


9 


43% 


1 0 


43% 


student exit interviews 


12 


57% 


1 3 


57% 


student portfolios 


8 


38% 


1 0 


43% 


graduates of the program 


20 


95% 


23 


100% 


teachers in field with whom the student worked 


21 


100% 


23 


100% 


school district administrators 


1 8 


86% 


20 


87% 


other sources 


7 


33% 


7 


30% 
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14 

19 colleges and universities (90%) with elementary programs 
and in 21 institutions (91%) with secondary programs. 
"Faculty evaluations" were used in 16 (76%) and 18 colleges 
(78%) with elementary and secondary programs^ respectively. 

Another internal source of assessm^ent was the 
"significance of an internal self study." Twenty 
institutions with elementary programs (88%) rated the self 
study as "highly" or "very highly" significant factor to 
program quality whereas 18 institutions with secondary 
programs (78%) ranked self study similarly. 

Table Three reveals a variety of student abilities that 
were used in determining program quality. Respondents were 
asked to indicate the presence or absence of student 
performance in regard to curricular features in assessing 
quality of teacher education programs. Such features 
pertained to abilities in organizing or formulating 
curriculum^ adapting content and instruction to learnerS/ 
assessing learning, and so on. The values ranged from 67% 
to 95% for elementary programs and 70% to 96% for secondary 
programs . 

Table Four depicts summative assessment data used to 
evaluate teacher education programs. Measures such as 
"student performance on exit exams" (43%), "student exit 
interviews" (57%), and "student portfolios" (38% lor 
e lemen tary programs and 43 % for secondary programs ) were 
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reported. Over 81% of the institutions used employment 
ratios data in their program evaluations. 

In non-tabular form^ it was reported that nine 
instit utionsr or almost 40%, had an average employment ratio 
below 80% for the last three years. Four institutions, or 
17%, had an empl oymen t rat io above 90% within the field of 
teaching. Also in non-tabular form, 16 institutions, or 
69%, had an average percentage for the last three years of 
greater than 90% of their students pass all four sections of 
the NTE. In contrast, 2 respondents, or 9%, rated the NTE 
"very highly" or "highly" as a valid indicator for 
contributing to the quality of teacher education programs. 

Additional assessment information was obtained by 
institutions from graduates, cooperating teachers, and 
school district administrators. In particular, 100% of the 
respondents reported obtaining feedback from cooperating 
teachers about the quality of their teacher education 
programs . 

Table Five refers to factors which c ontribute to 
overall quality of teacher education programs (Appendix C, 
Question 21). A 12 item Q-Sort type of response was used to 
a scert ai n t he pe rcept ions of school deans' and/or direct or s 
of teacher education programs. The beliefs were ranked 
according to the most important and least important factors 
that contributed to the overall quality of their teacher 
education programs. Mean responses were calculated based 



Table Five 

Perceived Importance of Factors That Contribute 
to Quality of Teacher Education Programs 



Mean 

1 .22 A) motivation and desire of the students 

1.17 B) program curriculum 

2.30 C) overall grade point average of the students 

1 .87 D) grade point average in the students' major 

1 .83 E) student teaching grade or performance (if on pass/fait) 

1,61 F) academic credentials of the education faculty 

1.09 G) academic quality of faculty instruction 

2.96 H) induction program within school districts 

2.91 I) evaluation processes from external agencies such as NCATE 

1.87 J) collaboration with school districts 

2.13 K) class size 

1.52 L) articulation across program components: i. e. general education, 
arts and sciences, professional education, field experience/ 
student teaching. 
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upon a five point scale. The four most important perceived 
factors were ranked: 1) academic quality of faculty 
instruction, 2) program curriculum, 3) motivation and desire 
of the students, and 4) articulation across program 
components. Two factors perceived as least important were 
"evaluation processes from external agencies such as NCATE" 
ranked 11 out of 12 and "induction programs" which ranked 
last . 

Table Six examines those features from external sources 
of assessment. Most notably were the NCATE, Middle States, 
and PDE evaluations of program features. 

It appeared that 10 institutions, or 43%, did not use 
the NCATE process as a means of assessing their teacher 
education programs. Of the 13 institutions that use the 
NCATE process, 6 institutions, or 46%, considered that 
process to contribute "very highly" or "highly" to the 
overall quality of teacher education programs. 

Regarding the significance of NCATE and Middle States 
evaluation processes, respondents' values for "highly" or 
"very highly" were 26% (6 schools), whereas the respondents' 
perceived opinions of their faculty ranged from 39% for 
NCATE and 2 6% for Middle States. 

In reference to the PDE evaluation, 17 institutions 
(74%) ranked the process "highly" or "very highly" whereas 
respondents' perceived opinion of their faculty were 
recorded slightly lower at 65% (15 institutions) . 
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Table Six 

Program Features from External Sources 
of Assessment 

How significantly does the NCATE process affect the overall quality of your teacher education 
programs? 

Respondent Respondent Perceived 
Opinion of Faculty 

Not applicable 10 (43%) 10 (43%) 

slightly 2 ( 9%) 4 (17%) 

moderately 5 (22%) 5 (22%) 

hi.ghly 2 ( 9%) 4 (17%) 
very highly 4 (17%) 

How significantly does the Middle States process affect the overall quality of your teacher 
education programs? 

Respondent ^^iculty Opinion 

Not applicable 1 ( 4%) ( 4%) 

slightly 7 (30%) 8 (35%) 

moderately 9 (39%) 8 (35%) 

highly 4 (17%) 5 (22%) 

very highly 2 ( 9%) 1 ( 4%) 

How significantly does the Pennsylvania Department of Education process affect the overall 
quality of your teacher education programs? 

Respondent Faculty Opinion 

Not applicable 

slightly 2 ( 9%) 4 (17%) 

moderately 4 (17%) 4 (17%) 

highly 9 (39%) 6 (26%) 

very highly 8 (35%) 9 (39%) 
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SUMMARY, CONCLUSIONS, AND RECOMMENDATIONS 



Summary 

1. Of the institutions offerring elementary and secondary 
programs, 17 institutions (81%) and 20 institutions 
(87^) for elementary and secondary programs place high 
priority upon S^:iiiiis^ti^:_AEi:iiiid£-.IEe5i scores and high 
schcol class standing for accepting students in 
Pennsylvania colleges and universities. Least used 
admission criteria were admissions interview and high 
school portfolios, respectively. (Table One) 

2. Regarding program evaluation, the two most frequently 
used features were faculty committee evaluations and 
student evaluations (90% and 91%, respectively), (Table 
Two) 

3. A variety of students* competencies were reported in 
assessing pro gram qual ity among the institutions 
surveyed. These competencies addressed curricular 
features such as abilities to organize and formulate 
curriculum, assess learning, adopt content and 
instruction to learners, and so on. Respondents used 
six out of s even compe tenci es over 90% of the t ime t o 
assess program quality. (Table Three) 

4 . Student performance on exit exams, student exit 
interviews, student portfolios and employment ratios 



were used to gather summative data on teacher education 
prcgram evaluation. However, the most effective source 
was found to be cooperating teachers. (Table Four) 

5. Four factors perceived as most important contributors to 
the overall quality of teachei education programs were 
1) academic quality of faculty instruction, 2) program 
curriculum, 3) motivation and desire of the students, 
and 4) articulation across program component . Two 
factors perceived as least important to quality were 
evaluations from external agencies (outside the system) 
and induction programs. None of the twelve items was 
perceived to be an unimportant contributor to program 
quality. (Table Five) 

6. NCATE, PDF, and Middle States were three external 
agencies used to evaluate program quality. Of the three 
external agencies, PDF was perceived to have the 
greatest effect on overall program quality. (Table Six) 

7. Six of 22 institutions (27%) considered the Middle 
States evaluation process to contribute "very highly^^ or 
"highly" to the overall quality of teacher education 
programs . 

8. Seventeen out of 23 institutions, or 74%, considered the 
PDF pro cess to contribute "highly" or "very highly" to 
the overall quality of teacher education programs. 
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9. The respondents felt their faculty did not perceive 

external agencies to be as important in contributing zo 
overall program quality as they did. 

Conclusions 

Measures for admissions into colleges and universities 
and into teacher education programs appear to be more 
quantitative than qualitative. The data suggest that 
scholastic aptitude test scores and high school class 
standing are the two largest determinants for accepting 
students in Pennsylvania colleges and universities. The 
selection criteria of portfolios and admissions interviews 
received a much lower rating even though the current 
literature suggests a greater emphasis on qualitative 
measures for selection to PA university and colleges. 

As they relate to assessing the quality of teacher 
education programs, the data suggest that internally derived 
factors are perceived to be more important than externally 
derived processes. The one major exception was the 
external-oriented agency of PA Department of Education. 

However, the data were inconsistent regarding the 
perceived importance that external agencies have upon the 
quality of teacher education programs. While the deans 
and /or directors of teacher education programs ranked the 
"evaluation processes from external agencies such as NCATE" 
as being, on the average, an unimportant factor with respect 
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CO their contributions to the quality of teacher education 
programs, they rated the ?DE evaluation process, an external 
agency, quite highly. It is possible that many officials at 
.institutions did not want to pay the NCATE fee or did not 
have the PDE in mind when answering this question. 

Although assessment of higher education is not mandated 
in Pennsylvania, the data suggest that assessment is 
certainly in the mainstream of thinking. .The results 
reflected a variety of assessment mechanisms on program 
evaluation in teacher education for students, faculty, and 
administration. The results showed a greater frequency with 
the more traditional, quantitative measures than qualitative 
indices. The next phase needs to focus on quality 

Recommendat ions 

1. Farther research is needed on ways to collect data on 
how teacher education programs are assessed. 

2. In-depth interviews should be conducted on what is and 
what is not helpful in the program evaluation process. 

3. A small research team needs to be formed to focus on and 
coordinate dissemination efforts based upon the results 
of pro j ect.5 and s tudie s . 

4. There needs to be stronger networking between program 
evaluation processes in teacher education programs and 
the Chapter 49 regulations that are currently being 
reviewed and reformulated. The Pennsylvania State Board 



of Education's Committee on Teacher Preparation and 
Certification has devised 14 policy questions to serve 
as focal points in revising these regulations. 
Because of differential expectations, it is absolutely 
essential that upcoming reform efforts be more strongly 
correlated bet we en higher education and basic e ducat ion . 
In order for more meaningful reform effort s to occur, 
school-university partnerships should be inextricably 
linked so that prospective teachers are inexorably 
linked to the tea ching -learn ing- as ses sment process. 
Accordingly, several paradigms of university teacher 
educators working in a participatory fashion with basic 
education teachers and schools need to be designed, 
field-tested, and revised. 

An assessment model of program evaluation for teacher 
education programs needs to be constructed, revised, and 
impleme nted . 
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Appendix A 



Part I: 



Program Evaluation 



Please indicate if the following factors figure into teacher education program evaluation. Select your an 
to eacn statement by placing checkmarks under the Elementary and/or Secondary program columns that 
applicable. 



1. A faculty committee or other governance body evaluates 
courses within the teacher education programs. 

I 

2. There is a set of policy guidelines for evaluating teacher 
education courses, 

3. Faculty evaluations are used to evaluate the quality of the 
teacher education programs. 

4. Student evaluations of courses are used to evaluate teacher 
education programs. 

5. Student grade distributions are used to evaluate the quality of 
teacher education programs. 

6. Measures of students' abilities in the following areas are used 
to evaluate the quality of teacher education programs: 

• organize or formulate curriculum 

• make subject matter comprehensible to learner 

• adapt content and instruction to learners 

• assess learning 

• use assessment to organize instruction 

• work with students of diverse backgrounds 

• set goals for continuing professional development 

7. Program evaluations incorporate data from the following areas: 

• graduation ratios 

• employment ratios 

• students' performances on exit exams other than the National 
Teacher Exam or the Pennsylvania Teacher Certification Test 

• student exit interviews 

• student portfolios 



Elem 


Yes 


No 























Sec 


Yes 


No 























8. Feedback from the following professionals is used to evaluate 
the quality of teacher education programs: 



• graduates of the program 

• teachers in the field with whom the student teachers worked 

• school district administrators 

Please state any other sources 

Part II: Admissions 
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Eiem 


Yes 


No 















Sec . 


Yes 


No 















Select your answers to each statement and place checkmarks under the Elementary and/or Secondary program 
columns that are applicable. Note: Some items in Question 10 ask for additional information. 



9. The institution has an open admissions policy. 

10. The following criteria are used to admit students to the 
university or college: 



Scholastic Aptitude Test combined score or American 
College Testing score [ minimum score accepted 



* high school class standing [ minimum rank 

* admissions interview 

* high school grade point average [ minimum grade 



* portfolios of student's high school work 

* other, please identify 



Elem 


Yes 


No 







Sec 


Yes 


No 









Eiem 


Sec 


Required 


No 


Optional 


Requirea 


No 


Optional 































































Which of the following are used to determine entrance 
into teacher education programs: 



* grade point average between 2.0 - 2.49 (^.0 scale) 

* grade point average of 2.5 or higher (4.0 scale) 

* approval by Dean of Education or appropriate official 

* interview committee 

* other, please identify 



Elem 


Incoming 


Transfers 



















Sec 


Incoming 


Transfers 
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Part 



Student Teaching Prerequisites 



Select your answers to each statement and place checkmarks under the Elementary ard/or Secondary program 
columns that are applicable. 

12. Student performance requirements for admission into student 
teaching include the following: 



* an overall grade point average of 2.5 or higher (4.0 scale) 

* at least a 2.0 or better in all required courses (4.0 scale) 

* at least a 2.0 in all professional education courses (4.0 
scale) 

* pre-student teaching field experiences 

* acceptable scores on a standardized test 

* other, please identify 



Elem 


Yes 


NO 























Sec 


Yes 


No 























Part IV: Institutional Information 

Select your answer to the question and place checkmarks in the appropriate column. 



13. What combined score on the Scholastic Aptitude Test is useH as 
a cutoff score for admission to the college or university? 



A 


B 


C 


D 


E 













A) not applicable 

B) above 1200 

C) 1 199-950 

D) 949-800 

E) open admissions 

Please identify other tests and cutoff scores used. 



14. What was the institution's total average percentage for the last 
three years of teacher education majors who have passed all 
four sections on the National Teacher Exam or the Pennsylvania 
Teacher Certification Tests? 



A) 
B) 

C) 
D) 
E) 



above 98% 
above 90% 
above 85% 
above 80% 
below 80% 



A 


B 


C 


D 


E 
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15. • Of those who graduate from your teacher education programs, 
what is the average percentage of employment within the field 
of teaching in the last three years? 



A 


B 


C 


D 


E 













A) 
B) 
C) 
D) 
E) 



above 95% 
above 90% 
above 85% 
above 80% 
below 80% 



Part V; 



Perceptions 



Select your answers to each statement and place checknnarks under the "Respondent" and "Perceived Faculty 
Opinions" columns. Note: This section involves your perceptions and your perceived opinions 
of your faculty. 



16. How significantly does the NCATE process affect the 
overall quality of your teacher education programs? 

A) not applicable 

B) slightly 

C) moderately 

D) highly 

E) very highly 

17. Hov/ significantly does the Middle States process affect 
the overall quality of your teacher education 
programs? 

A) not applicable 

B) slightly 

C) moderately 

D) highly 

E) very highly 

18. How significantly does the Pennsylvania Department of 
Education process affect the overall quality of your 
teacher education programs? 

A) not applicable 

B) slightly 

C) moderately 

D) highly 

E) very highly 



ResDondent 


A 


B 


C 


D 


E 













Perceivea 
Faculty Ooinion 


A 


B 


C 


D 


E 













A 


B 


\^ 


D 


E 













A 


B 


C 


D 


E 













A 


B 


C 


D 


E 













A 


B 


C 


D 


E 
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19. How significantly does an internal self-study process 
affect the overall quality of your teacher education 
programs? 



ResDondent 


A 


B 


C 


D 


E 













Perceived 
Faculty Opinion 


A 


B 


C 


D 


E 













A) not applicable 

B) slightly 

C) moderately 

D) highly 

E) very highly 



20. How would you rate the National Teacher Exam or 
Pennsylvania Teacher Certification Tests as a valid 
indicator of the overall quality of your teacher 
education programs? 



A) 
B) 
C) 
D) 
E) 



not applicable 
slightly 
moderately 
highly 
very highly 



A 


B 


C 


D 


E 













A 


B 


C 


D 


E 













21. Review the factors below with respect to their contributions to the quality of teacher education 

programs. Indicate the importance of each factor by rating it according to the following five point scale: 
( very important, (2) important, (3) moderately important, (4) slightly important, (5) not 
important. 



A) motivation and desire of the students 



B) program curriculum 



C) overall grade point average of the students 
grade point average in the students' major 

E) student teaching grade or performance (if on pass/fail) 

F) academic credentials of the education faculty 

G) academic quality of faculty instruction 

H) induction program within school districts 

I ) evaluation processes from external agencies such as NCATE 



J) collaboration with school districts 
K) class size 



L) articulation across program components: i.e., general education, arts and sciences, professions: 
education, field experience/student teaching. 
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Please use this space to Identify or further explain any other facets of how the education profession assesses or 
determines the quality of teacher education programs. 
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Letter to Pennsylvania 
Colleges and Universities From 
Pennsylvania Academy 



PErinSYLVATilA ACADEMY 

for the PROFESSION of TEACHIMQ 



6 August 1991 



1' 

Dear 2^, 

The Pennsylvania Academy for the Profession of Teaching is initiating a project 
to examine issues related to the assessment of program quality in teacher 
education. The intention of this project is to engage those involved in teacher 
preparation in a dialogue about these issues which will, in turn, contribute to 
improved program assessment policies and practices. 

Unlike a number of other states across the country Pennsylvania does not have a 
legislatively mandated program assessment ir place. This poses a significant 
opportunity for those of us in the field to generate program assessments which 
take into account the myriad factors that contribute to a program's quality and 
which are supportive of ongoing program quality enhancement. 

For a number of years program inputs have been used as the primary indices of 
program quality. Over the last two decades evidence of prospective teachers' 
ability to demonstrate specific discrete behaviors and/or knowledge recall have 
been folded into the program assessment mix. More recently, issues related to 
teachers' judgement, ability to apply knowledge and skills, and facility with 
negotiating the complexities of the classroom have emerged as salient dimensions 
of program assessment. Measuring these qualities and relating their presence in 
teacher candidates back to programs is nothing if not a complex tricky, and, as 
yet, unresolved business. Yet it seems compelling to persevere. 

The attached survey questionnaire is being sent to a sample of deans and 
directors of teacher education programs throughout the Commonwealth. Results of 
the survey will be used to inform the broader community of teacher educators 
about representative features of current program assessment practice. The 
results also will be used to stimulate the dialogue about program assessment. 
Your responses are critical to building a realistic picture of current practice 
in this area. Your assistance is greatly appreciated. The questionnaires are 
coded for identification so that follow-up telephone interviews might be done. 

Please complete and return the attached questionnaire in the enclosed stamped 
envelope by 5 September 1991. Feel free to call me if you have questions. 

Sincerely, 



Mary Gendernalik Cooper 
Project Director 



Enclosure 
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Appendix C 



Responses to Survey Instrunnent 



Part 



PROGRAM QUALITY ASSESSMENT SURVEY 
IN TEACHER EDUCATION IN PENNSYLVANIA 

Program Evaluation • Responses Appendix c 



Please indicate if the following factors figure ir.io teacher education program evaluation. Select your ansv/ers 
to each statement by placing checkmarks under the Elementary and/or Secondary program columns that are 
applicable. Percentages do not add to 100 in some cases because of no response or rounding off. 



1. A faculty committee or other governance bcdy evaluates 
courses within the teacher education programs. 

2. There is a set of policy guidelines for evaluating teacher 
education courses. 

3. Faculty evaluations are used to evaluate the quality of the 
teacher education programs. 

4. Student evaluations of courses are used to evaluate teacher 
education programs. 

5. Student grade distributions are used to evaluate the quality of 
teacher education programs. 

6. Measures of students' abilities in the following areas are used 
to evaluate the quality of teacher education programs: 

• organize or formulate curriculum 

• make subject matter comprehensible to learner 

• adapt content and instruction to learners 

• assess learning 

• use assessment to organize instruction 

• work with students of diverse backgrounds 

• set goals for continuing professional development 

7. Program evaluations incorporate data from the following areas: 

• graduation ratios 

• employment ratios 

• students' performances on exit exams other than the National 
Teacher Exam or the Pennsylvania Teacher Certification Test 

• student exit interviews 

• student portfolios 
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Feedback from the following professionals is used to evaluate 
the quality of teacher education programs: 
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• graduates of the program 

• teachers in the field with whom the student teachers worked 

• school district aaministrators 
Please state any other sources 

Part II: Admissions 

Select your answers to each statement and place checkmarks under the Elementary and/or Secondary program 
columns that are applicable. Note: Some items in Question 10 ask for additional information. 
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The institution has an open admissions policy. 
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The following criteria are used to admit students to the 
university or colleae: 
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11. Which of the following are used to determine entrance 
into teacher education programs: 



• grade point average between 2.0 - 2.49 (4.0 scale) 

• grade point average of 2.5 or higher (4.0 scale) 

• approval by Dean of Education or appropriate official 

• interview committee 

• other, please identify 
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Se'ect your answers to each statement and place checkmarks under the Elementary and/or Secondary program 
columns that are applicable. 

12. Student performance requirements for admission into student 
teaching include the following: 



• an overall grade psint average of 2.5 or higher (4.0 scale) 

* at least a 2.0 or better in all required courses (4.0 scale) 

• at least a 2.0 in ail professional education courses (4.0 
scale) 

* pre-student teaching field experiences 

• acceptable scores on a standardized test 

* other, please identify 
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Part IV: 



Institutional Information 



13. What combined score on the Scholastic Aptitude Test is used as 
a cutoff score for admission to the college or university? 

A) not applicable 

B) above 1200 

C) 1 199-950 

D) 949-800 

E) open admissions 

Please identify other tests and cutoff scores used. 
22=96% no 1=4% yes 

14. What was the institution's total average percentage for the last 
three years of teacher education majors who have passed all 
four sections on the National Teacher Exam or the Pennsylvania 
Teacher Certification Tests? 



A) 
B) 
C) 
D) 
E) 



above 98% 
above 90% 
above 85% 
above 80% 
below 80% 
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15. Of those who graduate from your teacher education programs, 
what is the average percentage of employment within the field 
*of teaching in the last three years? 
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B) 
C) 
D) 
E) 



above 95% 
above 90% 
above 85% 
above 80% 
below 80% 



Part V: Perceptions 

Select your answers to each statement and place checkmarks under the "Respondent" and "Perceived Faculty 
Opinions" columns. Note: This section involves your perceptions and your perceived opinions 
of your faculty. 



16. 



17. 



18. 



How significantly does the NCATE process affect the 
overall quality of your teacher education programs? 



A) 
B) 
C) 
D) 
E) 



not applicable 
slightly 
moderately 
highly 
very highly 
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How significantly does the Middle States process affect 
the overall quality of your teacher education 
programs? 



A) 
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How significantly does the Pennsylvania Department of 
Education process affect the overall quality of your 
teacher education programs? 
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19. 



20. 



21. 



How significantly does an internal self-study process 
*affect the overall quality of your tear" — -.- --^^-^ 
programs? 



A) not applicable 

B) slightly 

C) moderately 

D) highly 

E) very highly 



How would you rate the National Teacher Exam or 
Pennsylvania Teacher Certification Tests as a valid 
indicator of the overall quality of your teacher 
education programs? 
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A) not applicable 

B) slightly 
0) moderately 

D) highly 

E) very highly 

Review the factors below with respect to their contributions to the quality of teacher education 
programs. Indicate the innportance of each factor by rating it according to the following five point scale: 
(1) very Important, (2) important, (3) moderately important. (4) slightly important, (5) not 
important. Mean values are: 



1 .22 A) motivation and desire of the students 

1.17 B) program curriculum 

2.30 0) o.verall grade point average of the students 

1 .87 D) grade point average in the students' major 

1 .83 E) student teaching grade or performance (if on pass/fail) 

1 .61 F) academic credentials of the education faculty 

1 .09 G) academic quality of faculty instruction 

2.96 H) induction program within school districts 

2.91 I ) evaluation processes from extemal agencies such as NCATE 

1 .87 J ) collaboration with school districts 

2.13 K) class size 8 (35%) Responded 

15 (65%) No response 

1 .22 L) articulation across program components: i.e.. general education, arts and sciences, professional 
education, field experience/student teaching. 



Mean N=23 


Rank 


A 


1.22 


3 


B 


1.17 


2 


c 




in 


D 


1.87 


7.5 


E 


1.83 


6 


F 


1.fi1 




G 


1.09 


1 


H 


2.96 


12 


1 


2.91 


11 


J 


1.87 


7.5 


K 


2.13 


9 


L 


1.52 


4 



ERIC 



Appendix D 



Comments From Respondents 



53 



APPENDIX D 

Comment s Fr om Re spondent s 

Although we are not doing anything with portfolios at 
the moment they appear to have some value in the assessment 
process. The new tests being prepared by NTE are evidently 
going to be more performance based and that move should have 
a dramatic effect on program assessment as well as 
instructional practice. Frankly^ I would like to see some 
kind of intern experience similar to that experienced by 
doctors but the sheer magnitude of teacher education 
programs is a problem. 

At I UP we have been trying to evaluate cr it ica 1 
qualities of candidates. In our pre-teacher assessment 
program we are able to pinpoint these qualities very 
accurately. We also have depar tment ally developed 
evaluation processes at the junior level that includes an 
interview, specific grade requirements and in some cases a 
proficiency exam. 

I was highly displeased with the PA Dept. of Education 
review process. Surely there's a better way. 

We are in the process of developing a more formal 
procedure for evaluating and screening candidates throughout 
the four years of our undergraduate training program. I 
cannot share the specifics of this procedure at this time. 
Our Special Education /Elementar y Education Program has been 
evaluated recently by a National Inquiry Team sponsored by 
the Council for Exceptional Children (National Organization) 



Comparison with research findings 

Penn State is pleased that more of our students later 
earn doctorates than most places. This is fine for a 
research institution but it doesn't speak to the quality of 
the teacher preparation. We need to do more systematic 
assessment of employers of our students, but in the last ten 
years up to half our students annual ly accept posit ions 
outside Pennsylvania. I'd like to involve more teachers in 
evaluation, perhaps mentors in the induction program. 

The greatest problem is resources: time and staff. 
Teacher Ed, Programs don't budget for continuous program 
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evaluation. Currently, rising enrollment is strapping all 
of us just to meet student demand. 

The rest of the institution doesn't worry about 
evaluating teacher education programs because they already 
"know" that they are poor and not particularly worthy, 
though they have no real evidence of that . For example, we 
demonstrated that our Secondary Ed. Students* SATs and CPAs 
are equivalent to others in the University, but our 
c ol leagues ref us e to accept that bee ause they "know " that 
teacher ed. students are worse than others. 



Although we are working very diligently in improving 
our students* scores on the NTE , I believe that the 
mandatory taking of the NTE is expensive and is not an 
indicate ■ of how well a student will do as a teacher. I 
think the money the state spent on the test and the dollars 
spent by students is an exercise in futility. (As it 
pertains to making/creating good teachers) 



The determining factors in teacher education quality is 
the product, the student who becomes a stellar teacher. 

This past year we had a one hundred percent passing of 
the NTE. 



1) Institutional, state, and national assessment programs 
need to be integrated to avoid redundancy. 

2) Program graduates need to be assessed during their first 
2-3 years in the profession, 

3) SCDEs need to increase personnel and budgets to support 
more meaningful assessments, 

4) All teacher educators should be required to complete at 
j.east one graduate course on the assessment of teaching 
and teacher education , 



While the data you are collecting will be of some 
interest, I do not believe it addresses the fundamental 
problem in teacher education. That is, what empirical 
evidence is available (data collected from controlled 
experimentation) which demonstrates that the teaching of 
methods ~ eg. how to teach the elementary vs secondary 
child; how to teach social studies vs science; how to teach 
the gifted vs the retarded is effective in 
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effective teachers. We treat the methods approach 

as an (the way to prepare teachers) 

and have for the past 50 years. Yet there is research tc 
believe that all of these courses are based upon an 
erroneous assumption. That is^ we can design meaningful 
courses for teaching people classified on a single variable. 

When we feel all of us differ on several 

variables which be taken into account for evaluation 

ro be effective. We have addressed this issue and have some 
interesting results. Until we are willing to admit that our 
basic approach to teacher education has been faulty, it is 
real likely (unlikely) that significant changes will occur. 
Rather , what we will find (as we have in the past) is that 
the only significant predictor of teacher effectiveness are 
personal characteristics of the student and their academic 
performance not what they have learned in existing prof, 
educ. courses. Assessment, which is limited to products of 
flawed approaches to teacher education, will do little tc 
enhance the quality of teacher education. 

I will be happy to talk further with you if you are 
intere sted. 

Sincerely^ 



William Moore 
Bucknell Univ . 
(717) 524-1324 



